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Abstract Formalism based on GA is an alternative to distributed representation 
models developed so far — Smolensky's tensor product, Holographic Reduced Rep- 
resentations (HRR) and Binary Spatter Code (BSC). Convolutions are replaced by 
geometric products, interpretable in terms of geometry which seems to be the most 
natural language for visualization of higher concepts. This paper recalls the main 
ideas behind the GA model and investigates recognition test results using both inner 
product and a clipped version of matrix representation. The influence of accidental 
blade equality on recognition is also studied. Finally, the efficiency of the GA model 
is compared to that of previously developed models. 



1 Introduction 

Since the early 1980s a new idea of representing knowledge has emerged by the 
name of distributed representation. It has been the answer to the problems of recog- 
nition, reasoning and language processing — people accomplish these everyday 
tasks effortlessly, often with only noisy and partial information, while computational 
resources required for these assignments are enormous. To this day many models 
have been built, in which arbitrary variable bindings, short sequences of various 
lengths and predicates are all usually represented as fixed-width high dimensional 
vectors that encode information throughout the elements. In 1990 Smolensky lfT4l 
described how tensor product algebra provides a framework for the distributed rep- 
resentation of recursive structures. Unfortunately, Smolensky's tensor product does 
not meet all criteria of reduced representations as the size of the tensor increases 
with the size of the structure. Nevertheless, Smolensky and Dolan ITSl have shown. 
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that tensor product algebra can be used in some architectures as long as the size of 
a tensor is restricted. In 1994 Plate ifTSll worked up his Holographic Reduced Rep- 
resentation (HRR) that uses circular convolution and vector addition to combine 
vectors representing elements of a domain in hierarchical structures. Elements are 
represented by randomly chosen high-dimensional vectors. A vector representing 
a structure is of the same size as the vectors representing the elements it contains. 
In 1997 Pentti Kanerva flO' TTl introduced Binary Spatter Code (BSC) that is very 
similar to HRR and is often referred to as a form of HRR. In BSC objects are rep- 
resented by binary vectors and the boolean exclusive OR is used instead of convo- 
lution. The clean-up memory is an important part of any distributed representation 
model as an auto-associative collection of all atomic objects and complex statements 
produced by that system. Given a noisy extracted vector such structure must be able 
to recall the most similar item stored or indicate, that no matching object had been 
found. 

The geometric algebra (GA) model, which is the focus of this paper, is an alter- 
native to models developed so far. It has been inspired by a well-known fact, that 
most people think in pictures, i.e. two- and three-dimensional shapes, not by using 
sequences of ones and zeroes. As far as brain functions are concerned, geometric 
computing has been applied thus far only in the context of primate visual system 
(151, Chapters 1 and 2). 

In the GA model convolutions are replaced by geometric products and superpo- 
sition is performed by ordinary addition. Sentences are represented by multivectors 
— superpositions of blades. The concept of GA first appeared in the 19th century 
works of Grassmann and Clifford, but was abandoned for almost a century until 
Hestenes brought up the subject in |8| and 19]. The Hestenes system has recently 
found applications in quantum computation (Czachor et al. |[T|-||4l, Q), which ap- 
pears to be a promising leap from cognitive systems based on traditional computing. 

Section |2] of this paper recalls basic operations that can be performed on blades 
and multivectors, using the example Kanerva ifTTl gave to illustrate BSC. For fur- 
ther details on multivectors as well as interesting exercises the reader may refer to 
|l7l[T2|. Section [3] gives rise to discussion about various ways of asking questions 
and investigates the percentage of correctly recognized items under two possible 
constructions. Section[4]introduces measures of similarity based not on only the in- 
ner product of a multivector, but also on its matrix representation. Finally, Section 
|5] studies the influence of accidental blade equality on recognition and Section |6] 
compares the performance of the GA model with HRR and BSC. 



2 Geometric Algebra Model 

Distributed representation models developed so far were based on long binary or 
real vectors. However, most people tend to think by pictures, not by sequences of 
numbers. Therefore geometric algebra with its ability to describe shapes is the most 
natural language to mimic human thought process and to represent atomic objects 
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as well as complex sentences. Furthermore, geometric product of two multi vectors 
is geometrically meaningful, unlike the convolution or a binary exclusive OR oper- 
ation performed on two vectors. 

In this paper we consider the C£„ algebra generated by orthonormal vectors bi = 
{0, . . . , 0, 1 , . . . , 0} for / S {!,...,«}. The inner product used throughout the paper 
is an extension of the inner product (-I-) from the Euclidean space M". For blades 
X<^> = xi A ■ ■ ■ Axii and y</> — yi A ■ ■ ■ Ayi the inner product • : C£„ x C£„ ^ M is 
defined as 



(X. 



(X. 



{xi\yi) {xi\yi-i) 
{x2\yi) {x2\yi-i) 

{xk\yi) {xk\yi-i) 

foik^l 



{xi\yi) 
(xilyi) 

{xk\yi) 



for k = L 



(1) 



(2) 



and is extended by linearity to the entire algebra. 

Originally, the GA model was developed as a geometric analogue of BSC and 
HRR and was described by Czachor, Aerts and De Moor in |[T| and lIU. Before 
switching from geometric product to BSC and HRR one has to realize, that geomet- 
ric product is a projective representation of boolean exclusive OR. Let xi ...x„ and 
yi ...y„he binary representations of two «-bit numbers x and y and let — Cx 



.bf^" andcv = Cv,...,„ = b\^ , 



. . bl" be their corresponding blades, b^ being equal 
to 1. The following examples show that geometric product of two blades Cx and Cy 
equals, up to a sign, Cx(sy 



bibi 
bibn 
bnb\ 



C10...0C10...0 = 1 = C0...0 = C(io.. . 0)0(10.. .0), 

C10...0C110...0 = bibibi = b2 = coio...o = C(io...o)©{iio...o)j 

C110...0C10...0 = bib2bi = -b2b\b\ = -b2 

-C010...0 = -C(iio...o)e(io...o) = (^l)^C(iio o)®(io...o)' 



the number D being calculated as follows 

= yi (JC2 H +3'2(JC3 H l-x„) H ^yn-ix„ 



- Y^ykXi- 

k<l 



(3) 
(4) 

(5) 



(6) 



The original BSC is illustrated by an example taken from lflTl l4l — atomic ob- 
jects are represented by randomly chosen strings of bits, "0" is a componentwise 
addition mod 2 and "ffl" represents a thresholded sum producing a binary vector — 
the threshold is set at one half of sentence chunks and a random string of bits is 
added in case of an even number of sentence chunks to break the tie. The encoded 
record is 



PSmith — {name (B Pat) ffl {sex®male) ffl (flge©66), 



(7) 



and the decoding of name uses the involutive nature of XOR 
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Pat' = name © PSmith 

~ name © \{name © Pat) ffl {sex © male) ffl (age © 66)] 

= Pat ffl [name © se;ic © male) ffl {name © age © 66) 

= Paf ffl noise Pat . (8) 

In order to switch from BSC to HRR, the map described in ||4J is used. 

In the GA model, roles and fillers are represented by randomly chosen blades 

PSmith — name * Pat + sex * male + age * 66 . (9) 

The "+" is an ordinary addition and written between clean-up memory items 
denotes the geometric product — this notation will be traditionally omitted when 
writing down operations performed directly on blades and multivectors. The "+" 
written in the superscript denotes the reversion of a blade or a multivector. The 
whole record now corresponds to a multivector 

PSmith = Ca^...a„Cxi...x„ +ct^...b„Cy^...y„ +Qi...c„Cj,...^„, (10) 

and the decoding operation "jj" of name with respect to PSmith is defined as follows 

PSmith name — name^ ^PSmith (11) 

= Cx i Ca(Qbmy i Ca(ScS)z (12) 

= faf + noise. (13) 

It remains to employ the cleanup memory to find the element closest to Pat' — 
similarity is computed by the means of the inner (scalar) product. When using the 
decoding symbol "(1" we assume that the reader knows which model is used at the 
time. Therefore, there will be no variations of the "ft" symbol in BSC, HRR or in 
two possible GA models (depending on the way of asking a question). 

For an actual example let us choose the following representation for roles and 
fillers of PSmith 



■ fillers (14) 



Pat -- 


= CQoioo, 


male = 


= CQOlll, 


66 = 


= ciiooo, 


name 


= CQOOIO 


sex 


= Ciiioo 


age 


= Cloool 



■roles (15) 



The whole record then reads 
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PSmith = name * Pat + sex * male + age * 66 

= CooOIOCOOlOO + C11100C00111 +C10001C11000 
= — COOl 10 + CllOl 1 + coiool ■ 

The decoding of PSmith'?, name will produce the following result 

name^ * PSmith = cqoioo + C11001 -coioil 
~ Pat + noise ~ Pat' . 



(16) 



(17) 



At this point, inner products between Pat' and the elements of the clean-up mem- 
ory need to be compared. Item in the clean-up memory yielding the highest inner 
product will be the most likely candidate for Pat 



{Pat\Par 
(male\Pat' 
{66\Par 
{name\Pat' 
{sex\Pat' 
{age\Pat' 
{PSmith\Par 



— cooloo • (cooioo + ciiooi — coiou) = 1 7^ 0, 

= 0, 

= 0, 

-0, 

-0, 

-0, 

= 0. 



A question arises as to how to extract information from a multivector - 
a question be asked on the left-hand-side of a multivector 



or the right-hand-side 



name * PSmith, 



PSmith * name. 



(18) 
(19) 
(20) 
(21) 
(22) 
(23) 
(24) 

should 
(25) 
(26) 



Furthermore, should we use name or rather name^l Since we can ask about both 
the role and the filler, we should be able to ask both right-hand-side and left-hand- 
side questions according to the principles of geometric algebra. Such an approach, 
however, would make the rules of decomposition unclear, which is against the phi- 
losophy of distributed representations. The problem of asking reversed questions on 
the appropriate side of a sentence is that we should be able to distinguish roles from 
fillers. This implies that atomic objects should be partly hand-generated, which is 
not a desirable property of a distributed representation model. If we decide that a 
question should always be asked on one fixed side of a sentence, there is no point in 
reversing the blade since there is no certainty that the fixed side is the appropriate 
one. Independently of the hand-sidedness of questions, in test results the moduli of 
inner products are compared instead of their actual (possibly negative) values. For 



right-hand-side questions we can reformulate Equations ( 1 1 
way 



(1 3 1 in the following 
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PSmith tJ name = PSmith * name 



■yi---yn 



+ 



(27) 



'-Cl...C„t-Zl...Z„ J <-«!...«„ 

±Pflf + noise = Pat' . 



(28) 
(29) 
(30) 



The decoding of PSmith's name will then take the form 



PSmith * name 



cooioo — ciiooi — coioii =Pat', 



(31) 



resulting in KPS'm/f/zlPflf')! = | — 1| = 1. We will study the effects of asking ques- 
tions in various ways in the next Section. 



3 Recognition 

Before we investigate the percentage of correctly recognized items, we need to intro- 
duce the following definitions. Let S and Q denote the sentence and the question re- 
spectively. Let be the set of all clean-up memory items A for which {S ^Q\A) ^ 0. 
We will call a set of potential answers. Let m — max{| (5 (J 21^) I • ^4 G ^} and 
r = {A € ^ : 1(5 tt 21^4)1 = m). A pseudo-answer is an answer belonging to set 
T but different than the correct answer to 5 jj 2 — even if the difference is only 
in the meaning and not in the multivector Of course, set T might also include the 
correct answer — therefore, it is called the set of {pseudo-)correct answers and is 
actually the set of answers leading to the highest modulus of the inner product. We 
assume that a noisy statement has been recognized correctly if its counterpart in the 
clean-up memory is among the (pseudo-)correct answers. 

There are some doubts concerning how the sentences should be built — Plate lfT3l 
adds an additional vector denoting action id (usually a verb) to a sentence, e.g. 



where "®" denotes circular convolution. We will distinguish between two types of 
sentence constructions 

• Plate construction, e.g. eat + eatagt* Mark + eatf,i,j*theFish, 

• agent-object construction, e.g. eatagt * Mark + eatghj * theFish. 

The agent-object construction will often be denoted as "A-O" for short, especially 
in table headings. Preliminary tests conducted on the GA model were designed to 
investigate which type of construction suits GA better The vocabulary set and the 
sentence set for these tests are included in Table [T| The sentence set is especially 
filled with similar sentences to test sensitivity of the GA model to confusing data. 
Each sentence carries a number (e.g. "(3a)") to make further equations more com- 
pact and readable. 



{ eat + eatagt ® Mark + eatobj ® theFish) /V3, 



(32) 
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Table 1 Contents of the clean-up memory used in tests described throughout this paper. 



Number of 
blades 


Contents 


i 


A total of 42 atomic objects: 19 fillers, 7 single-feature roles and 8 double-feature 
roles 


z 


(la) biteagt * Fido -!- biteoiyj * Pcit 


3 


(3a) seeagt * John + see„tj*{^3i) 

{P Smith) name * Pat + sex * male + age * 66 


4 


(lb) bite„gt * Fido + bite„i,j * PSmith 
(2c) flee,ig, * PSmith + fleeobj * Fido 
(4a) cause agt *(la)+cause„i,j*(2a) 


5 


(3b) see^gt* John + seeoijj*(Vo) 
(5 a) see agt * John + see 


6 


(4c) causeag,*(Vo)+cause„i,j*(2a) 


7 


(DogFido) class * animal + type * dog + taste * chickenl ike 

+name * Fido + age * 7 + sex * male + occupation * pet 


8 


( 1 c) bitCagt * DogFido + bite„ijj * Pat 
(2b) fleeagt * Pat + flee„bi * DogFido 
(4b) causeagi*iib)+cause„i,i*{2c) 


9 


(3c) seeagt*John + see„i,j*(lc) 
(5b) seeagi * John + see„i,j*{Ah) 


10 


(Id) bite agt * DogFido + biteotj * PSmith 
(2d) flee,igt * PSmith + flee„i,j * DogFido 


11 


(3d) seeagt*John + see„i,i*(id) 



(*' Each sentence carries a number (e.g. "(3a)") to make further equations more readable. 



3.1 Right-hand- side questions 

In the previous Section we commented on the use of reversions and the choice of 
the side of a statement that a question should be asked on. The argument for right- 
hand-side (generally: fixed-hand-side) questions without a reversion was that rules 
of decomposition of a statement should be clear and unchangeable. However, the 
use of right-hand-side questions poses a problem best described by the following 
example. Let the clean-up memory contain seven roles and fillers 

secagt = cooioi, John = cqoioi, 

seeobj = coioio, P^^t = cioooo, .^3) 

biteagt = ciouo, Fido = cioooi, 

bite„bj — cooooi, 

and two sentences mentioned in Table [T] 

(la) biteagt * Fido + biteobj* Pat ^CQoiii-cioQoi, (34) 
(3a) seeagt * John + seeobj* (la) = -CQoom-coim-cnon- (35) 

Let us now ask a question (3a) seeobj — (3a) * seeobj- The decoded answer 
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(3a) ^seeohj = (-cooooo-coiioi -ciioii)coioio 

= — COIOIO + C00111 +C10001 

= noise + biteagt * Fido — bitef,hj * Pat 



(36) 
(37) 



results in one noisy chunks and two chunks resembling sentence (la) but having a 
partially different sign than the original (la). Furthermore, 

(la)- ((3a)*5ee„i^) = (cooiii -cioooi) ■ (-coioio+cooiii +cioooi) 

= cooi 1 1 • cooi 1 1 — cioooi • cioooi (38) 
= -1 + 1=0. (39) 

Such a situation would not have happened if we asked differently 



'obj 



*(3a), 



(40) 



since see^i_^jseeai,j = 1 for normalized atomic objects. 

The similarity of (la) and (3a)*seeot,j equals zero because the nonzero similari- 
ties of blades (i.e. Is) belonging to these statements cancelled each other out. Can- 
cellation could be most likely avoided, if sentence (la) had an odd number of blades. 
This observation has been backed up by test results comparing the performance of 
Plate construction and the agent-object construction. 

As expected, the agent-object construction seems to work better for sentences 
from which a rather simple information is to be derived, e.g. PSmith jj name or 
(5a) jj secobj (Figures [T| and |2]respectively). However, when the information asked 



[%] recognition 
100%- 

50% 
10% 



o Plate construction 

• Agent-object construction 

QUESTION: PSmith (t name 
ANSWER: Pat 
NUMBER OF TRIALS: 1000 
(right-hand-side questions) 



H 1 1 1 1- 



5 10 15 20 

Fig. 1 Recognition test results for PSmith J name. 



(«) 

N 


(b) 
Plate 


A-0 


l(^)-(^-)l 


4 


8.9% 


12.6% 


3.7% 


5 


36.3% 


43.1% 


6.8% 


6 


65.6% 


70.0% 


4.4% 


7 


85.5% 


87.0% 


1.5% 


8 


94.2% 


94.6% 


0.4% 


9 


96.0% 


96.0% 


0.0% 


10 


99.0% 


99.0% 


0.0% 


11 


99.4% 


99.4% 


0.0% 


12 


99.9% 


99.9% 


0.0% 


13 


99.9% 


99.9% 


0.0% 


14 


99.9% 


99.9% 


0.0% 


15 


100.0% 


100.0% 


0.0% 


16 


100.0% 


100.0% 


0.0% 


17 


100.0% 


100.0% 


0.0% 


18 


99.9% 


99.9% 


0.0% 


19 


100.0% 


100.0% 


0.0% 


20 


100.0% 


100.0% 


0.0% 



Geometric Algebra Model of Distributed Representations 



9 



[%] recognition 



100% 



50% 



10% 



o Plate construction 

• Agent-object construction 

QUESTION: (5a) B seeag, 
ANSWER: John 
NUMBER OF TRIALS: 1000 
(right-hand-side questions) 



H 1 1 1 1- 



5 10 15 20 

Fig. 2 Recognition test results for (5a) jj seeagt- 



[%] recognition 



100% 



50% 



10% 



o Plate construction 

■ Agent-object construction 



° o o ° o 



O - o O o 



QUESTION: (5a) S see„hj 
ANSWER: (4a) 
NUMBER OF TRIALS: 1000 
(right-hand-side questions) 



H 1 1 1 1- 



5 10 15 20 

Fig. 3 Recognition test results for (5a) jj seeobj- 



(«) 

N 


(b) 
Plate 


A-0 


\(b)-(c)\ 


4 


0.6% 


4.4% 


3.8% 


5 


3.3% 


18.5% 


15.2% 


6 


15.3% 


45.8% 


30.5% 


7 


47.8% 


76.4% 


28.6% 


8 


72.2% 


88.9% 


16.7% 


9 


86.4% 


94.3% 


7.9% 


10 


95.1% 


98.2% 


3.1% 


U 


97.7% 


98.9% 


1.2% 


12 


98.1% 


99.2% 


1.1% 


13 


99.1% 


99.6% 


0.5% 


14 


99.9% 


99.9% 


0.0% 


15 


99.6% 


99.9% 


0.3% 


16 


100.0% 


100.0% 


0.0% 


17 


100.0% 


100.0% 


0.0% 


18 


100.0% 


100.0% 


0.0% 


19 


100.0% 


100.0% 


0.0% 


20 


100.0% 


100.0% 


0.0% 



(a) 
N 


(*) 
Plate 


(c) 
A-O 


m-{c)\ 


4 


26.4% 


32.2% 


5.8% 


5 


39.2% 


44.2% 


5.0% 


6 


41.8% 


48.6% 


6.8% 


7 


47.3% 


50.1% 


2.8% 


8 


61.4% 


51.7% 


9.7% 


9 


61.2% 


50.6% 


10.6% 


10 


64.0% 


49.5% 


14.5% 


11 


65.4% 


49.3% 


16.1% 


12 


63.2% 


48.7% 


14.5% 


13 


64.3% 


49.9% 


14.4% 


14 


66.4% 


50.4% 


16.0% 


15 


64.9% 


48.8% 


16.1% 


16 


66.9% 


49.8% 


17.1% 


17 


67.3% 


53.3% 


14.0% 


18 


66.4% 


49.4% 


17.0% 


19 


65.4% 


48.8% 


16.6% 


20 


65.2% 


51.1% 


14.1% 



was more complex, e.g. (5a) jJ secahj, the Plate construction seemed more appropri- 
ate (Figure |3]l. This might be so for at least two reasons: 

• some of the blades belonging to the answer of (5a) jl see^hj appear in numer- 
ous entries in the cleanup memory listed in Table [T] causing the GA model to 
misinterpret the answers it receives after the inner products have been computed, 
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[%] recognition 



100% 



50% 



10% 



o Plate construction 

• Agent-object construction 

QUESTION: (Ih) t bite 
ANSWER: PSmith 
NUMBER OF TRIALS: 1000 
(right-hand-side questions) 



10 



15 



H 1 1 1 1— > 

20 



(«) 

N 


(b) 
Plate 


A-O 


\(b)-(c)\ 


4 


22.5% 


28.6% 


6.1% 


5 


36.3% 


44.6% 


8.3% 


6 


54.6% 


66.5% 


11.9% 


7 


72.6% 


79.5% 


6.9% 


8 


85.8% 


89.4% 


3.6% 


9 


93.5% 


96.1% 


2.6% 


10 


96.4% 


97.2% 


0.8% 


U 


98.0% 


98.3% 


0.3% 


12 


99.4% 


99.5% 


0.1% 


13 


99.5% 


99.7% 


0.2% 


14 


99.7% 


99.8% 


0.1% 


15 


100.0% 


100.0% 


0.0% 


16 


99.9% 


100.0% 


0.1% 


17 


100.0% 


100.0% 


0.0% 


18 


100.0% 


100.0% 


0.0% 


19 


99.9% 


100.0% 


0.1% 


20 


100.0% 


100.0% 


0.0% 



Fig. 4 Recognition test results for (lb) jj bite,, 



• the number of blades in (5a) secobj is uneven when using Plate construction, 
hence the possibility that blades' similarities cancel each other out is smaller than 
in case of the agent-object construction — such hypothesis would be backed up 
by test results depicted in Figure]?] 

These hypotheses led to a conclusion that perhaps a random blade should be added 
to those sentences that have an even number of blades, similarly to BSC. 
A correct answer might not be recognized for two reasons: 

• the correct answer has an even number of blades and their similarities cancelled 
each other out completely because of having opposite signs - hence such an an- 
swer does not even appear within the set of potential answers, 

• there are some pseudo-answers leading to a higher inner product because the 
similarities of blades of a correct answer cancelled each other partially. 

Adding random extra blades that make the number of blades in a multivector odd 
(for short: adding blades) is a solution to the first reason why a correct answer is 
not recognized. Further, an odding blade acts as a distinct marker belonging only to 
one sentence (for sufficiently large data size) distinguishing it from other sentences, 
unlike the extra blade representing action in Plate construction which may appear 
in numerous sentences. Unfortunately, to address the second problem, we need to 
employ some other measurement of similarity than the inner product. We will show 
in Section]?] that Hamming and Euclidean measures perform very well in that case. 

Observation of preliminary recognition test results led to a conclusion, that sen- 
tences with an even number of blades behave quite differently than sentences with 
an odd number of blades. In the following tests we inspected the average number 
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of times that blades' similarities cancelled each other out completely during the 
computation of similarity via inner product. 

The complete cancellation of similarities takes place only when exactly half of 
blades of the correct answer carry a plus sign and the other half carry a minus sign. If 
the correct answer has 2K > 2 blades, then the probability of exactly half of blades 
having the same sign is 

^ (41) 

under assumption, that the sentence set is chosen completely at random without 
the interference of the experimenter. Figures |5] through |7] show three examples of 
questions yielding an even-number blade answer and the average number of times 
their blades' similarities cancelled each other out completely. 



(42) 



3.2 Appropriate-hand-side reversed questions 

Let us recall some roles and fillers 

secagt = cooioi, John = cqoioi, 

secobj = coioiO) Pat — cioooo, 

biteagt = cioiio, Fido = cioooi, 
bite„bj = cooool, 

as well as two sentences mentioned in Table [T] 

( 1 a) bitCagt * Fido + biteotj * Pflf = cqoi 1 1 - cioooi , (43) 
(3a) iee„g,*yo/!« + 5ee„i,,*(la) = -cooooo-coiioi -cuoii- (44) 

The answers to questions (3a) jj seCobj and (3a) jj John should be computed in dif- 
ferent ways 

(3a) see„hj = see+f,. * (3a) « (la), (45) 
(3a) ^John — {3a.)* John^ « secagt- (46) 

We will concentrate only on the first question 

■'^^rJfoj * (3a) = Co[ojo(-cooooo-coiioi -cnon) 

= C01010(C00000 + C01101 +C11011) 

= coioio + cooi 1 1 — cioooi (47) 
= noise + (la) = (la)'. (48) 

Only two elements of the clean-up memory are similar to (la)' 
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avg % of correct answers 
belonging to potential answers 



100% 



50% 



10% 



QUESTION: (5a) (I seeobj 
ANSWER: (4a) - 4 blades 
NUMBER OF TRIALS: 1000 



1-^/2" =0.625 

Agent-object construction 
(right-hand-side questions) 



10 



15 



-I — I — I — I — I — I — I — I — I — I — I — I — I — I — I — I- 



20 



25 



30 



35 



-N 



Fig. 5 The average number of times a correct answer appears within the set of potential answers 
((5a) jt seeobj). 



avg % of correct answers 
belonging to potential answers 



100% 



50% 



10% 



QUESTION: (5b) i see„i,j 
ANSWER: (4b) - 8 blades 
NUMBER OF TRIALS: 1000 



1 - ^ = 0.726563 

■ Agent-object construction 
(right-hand- side questions) 



10 



15 



-I 1 1 1 1 1 1 1 1 1 1 1 1 1 1 H 



20 



25 



30 



35 



■N 



Fig. 6 The average number of times a correct answer appears within the set of potential answers 
((5b) J seeobj). 



avg % of correct answers 
belonging to potential answers 



100% 



50% 



10% 



QUESTION: (3d) i see„tj 
ANSWER: (Id) - 10 blades 
NUMBER OF TRIALS: 1000 



/2^" = 0.753906 



Agent-object construction 
(right-hand-side questions) 



10 



15 



H 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1- 



20 



25 



30 



35 



-N 



Fig. 7 The average number of times a correct answer appears within the set of potential answers 

((3d) J seeobj). 
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\{see„bj\{l^)')\ = l, (49) 
|((la)|(la)')| = |(cooiii -cioooi)- (coioio + cooiii -cioooOl 

= |C00111 -COOIII +C10001 • ClOOOll 

= 1-1-11=2, (50) 

where secobj is similar to the noise term only by accident. 

Asking reversed questions on the appropriate side has one huge advantage over 
fixed-hand- side questions: no similarities cancel each other out neither completely 
nor partially while similarity is being computed, hence there is no need for adding 
odding vectors. For small data size blades may cancel each other out at the moment 
a sentence is created. Nevertheless, in all cases recognition will quickly reach 100% 
and the only problem that might appear is that several items of the clean-up memory 
might be equally similar. 



4 Other measures of similarity 

The inner product is not the only way to measure the similarity of concepts stored 
in the clean-up memory. This Section comments on the use of matrix representa- 
tion and its advantages in the unavoidable presence of similarity cancellation and 
many equally probable answers. We will show, that comparison by Hamming and 
Euchdean measures gives promising results such cases. 



4.1 Matrix representation 

Matrix representations of GA, although not efficient, are useful for performing 
cross-checks of various GA constructions and algorithms. An arbitrary n-bit record 
can be encoded into the matrix algebra known as Cartan representation of CUfford 
algebras as follows 

^2* = (7i <?)••• Of^iO (72 (51) 

^ V ' ' V ' 

n-k k-l 

b2k-i = 0\®---®a\®a2,®\(^---(^\, (52) 



n-k k-l 



using Pauli's matrices 



c^i = l?J). 02=r. A, o,= [l^^ ). (53) 



Let us once again consider the roles and fillers of PSmith 
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Pat 


- CQoioo, 


male 


- CQOlll, 


66 = 


= ciiooo, 


name 


= coooio 


sex 


= ciiioo 


age 


= cioooi 



as described in Section[2] Their explicit matrix representations are 



Agnieszka Patyk 
(54) 

(55) 
(56) 



Pat — cooioo ~bi — a\®0\®a\®ai®\, 
male — cooiii — b^b^bs 

= (ai (g) (Ti (Xi CJl (gi (T3 (gi 1 ) (C7l (8) C7i (g) (Ji (g) (J2 (X" 1 ) (<7l (8) CTi (g) (J3 (g) 1 (Ki 1 ) 
= CTig)C7lg)CT3g)(-/(Tl)g)l, (57) 
66 = CiiooO = bib2 = (C7i g) C7i g) ai (g dl (g CJ3)(c7i (g C7i (g) C7i (g dl (g C72) 

= l«)l«)lg)lg)(-/(7i), (58) 
name — cqooio = ^4 = Ci g) C7i g) <Ti g) C72 g) 1 , (59) 
sex = ciiioo = bib2b3 

= (ai g) (Tl (g (Jl (g (Ji (g (T3)(0'l g) Ci g) ffl (g (Jl (g (T2)(<7l g) CTi g) ffi g) (T3 (g 1 ) 

= ai g)ai (g(j3 (g 1 (g (-/(Ji), (60) 
age — Cloool — bibs — (cJi g) ffi g) ffi g) CTi g) C73)(c7i g) CTi (g C73 g) 1 (g 1) 

= l«)l(g(-/a2)(gfflg)C73. (61) 

Figure [8] shows six blades making up PSmith for n = 5 and Figure |9] shows the 
matrix representation of PSmith for n E {6,7}, black dots indicate nonzero matrix 
entries. 




PSmith 



coooiocooloo 
(name * Pat) 

Fig. 8 PSmith and its blades for fi = 5. 



ciuoocooiu 
(sex * male) 



cioooicuooo 
(age * 66) 



The regularity of patterns placed along the diagonals is not an accident. Consider 
Cartan representation of blades b2ic and b2k-i — the shortest sequence of n — k 
CTi's will occur for k = [j] (in other words, in blade b„). Therefore each blade 
b\,...,bn has at least [jj of ai's placed at the beginning of the formula describing 
its representation. Hence, there are exactly 2 Li J "boxes" of patterns placed along one 
of the diagonals, each one of dimensions 21^21 x 2^^\ To extract a part individual 
for a given blade, one needs to consider only the last \j] + 1 of a's or unit matrices 
belonging to its representation — the extra Gi bearing the number « — \^~\ is needed 
to preserve the direction of the diagonal — either "top left to bottom right" or "top 
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(a) n = 6 



name = 001001 Paf = 000001 
ie!c=110011 mo/e= 110101 
age = 010001 66 = 001011 























\ 



(b) H = 7 

name = 1010100 Par = 1000001 
i£ar = 0000110 ma/e= 1010110 
age = 0011010 66 = 0000100 



Fig. 9 An example of matrix representation of PSmith for n e {6, 7}. 



LHS-signature 
RHS-signature 



right to bottom left". If c ~ ba^ ■ ■ ■ ba„, is a blade representing an atomic object in 
the clean-up memory, say Pat, then such an object has a "top left to bottom right" 
orientation if and only if m = O(mod 2). Therefore we can reformulate equations 



( 5 1 1 and ( 52 1 in the following way 

b2k^Oi(E)-- ■ (g) <Ji^ (g) 02 ® 1(8) --(8)1 , (62) 

= ffi <8) • • • <8) (Ji (8 (73 (8 1 (8 • • • 8) 1 . (63) 



\j\-k+l 



k-l 



Consider once again the representation of PSmith depicted in Figure |9]3. To dis- 
tinguish PSmith from other object we only need to store two of its "boxes" — each 
"box" lying along a different diagonal, Figure[9]3 shows such two parts. We will call 
the two different "boxes" left-hand-side signatures and right-hand-side signatures 
depending on the comer the diagonal is anchored to at the top of the matrix. It is 
worth noticing that signatures for n — 2k — I and « = 2fe are of the same size, which 
causes some test results diagrams to resemble step functions. 

Note that the use of tensor products in GA bears no resemblance to Smolensky's 
model, as the rank of a tensor does not increase with the growing complexity of a 
sentence. 
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The first most obvious method of comparing two matrices or their signatures would 
be to compute the number of entries they have in common and the number of 
entries they differ by. Let X = [x^] and Y = [y,j] be signatures of matrices, i.e. 
je{l,...2ril+i},7e{l,...2ril}.Let 

'^{xij,yij) = 1 - c{xij,yij). (65) 

Now let us count the number of common points and uncommon points 

C(X,y)=^c(xy,3;,-,), (66) 
'■i 

U{X,Y)=Y,u{xij,yij). (67) 

Finally, the Hamming measure for comparing the signatures of matrices computes 
the ratio of common and imcommon points 

H{X,Y)=[W9) iff^(^'i')^0. (68) 
I oo otherwise. 

Such a measure of similarity is fairly fast to calculate since it does not involve com- 
puting any mathematical operations except addition and the final division of C(X,Y) 
andU(X,Y). 



4.3 The Euclidean measure of similarity 

The second most obvious method for computing matrix similarity is via Euclidean 
distance. Again, let X = and Y = [y,j] be signatures of matrices for ; g 

{i,...2r?i+i},ye{i,...2r?i}.Let 



EiX,Y)=n^/^^-^\ (69) 
y oo otherwise. 

This kind of measure uses more mathematical operations requiring greater time to 
compute — the modulus of a complex number, multiplication and the square root. 
The Hamming measure involved calculating only addition and the ratio of common 
and uncommon points. Calculating the ratio in both measures results in those mea- 
sures taking on a role of "probability" that the matrices are ahke rather than describ- 
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QUESTION: (4a) jj causeobj 
ANSWER: (2a) 
NUMBER OF TRIALS: 500 

■ Inner product 

° Euclidean measure 

• Hamming measure 

(right-hand-side questions) 



Agent-object construction. 
[%] recognition 



100% 



50% 



10% 



10 



15 



H 1 1 1 M 

20 



N 



N 


Hamming 


Euclidean 


Inner pr. 


4 


16.2% 


31.4% 


20.2% 


5 


42.8% 


51.0% 


42.8% 


6 


37.0% 


43.8% 


45.2% 


7 


45.2% 


47.6% 


48.6% 


8 


44.4% 


45.8% 


47.6% 


9 


48.4% 


48.6% 


48.8% 


10 


46.6% 


47.0% 


47.0% 


11 


47.8% 


47.8% 


47.6% 


12 


51.0% 


51.6% 


51.6% 


13 


50.4% 


50.4% 


50.4% 


14 


51.6% 


51.6% 


51.6% 


15 


45.2% 


45.2% 


45.2% 


16 


52.6% 


52.6% 


52.6% 


17 


51.6% 


51.6% 


51.6% 


18 


50.4% 


50.4% 


50.4% 


19 


52.4% 


52.4% 


52.4% 


20 


50.2% 


50.2% 


50.2% 



Agent-object construction with odding blades. 
[%] recognition 



100% 



50% 



10% 



-H 1 1 1 1- 



-N 



N 


Hamming 


Euclidean 


Inner pr. 


4 


12.6% 


31.2% 


14.0% 


5 


42.2% 


70.0% 


25.6% 


6 


45.8% 


75.8% 


36.8% 


7 


83.4% 


89.8% 


57.4% 


8 


91.0% 


96.2% 


74.4% 


9 


97.0% 


98.4% 


88.4% 


10 


98.6% 


99.2% 


94.8% 


11 


99.8% 


99.8% 


97.8% 


12 


99.6% 


100.0% 


99.2% 


13 


100.0% 


100.0% 


99.4% 


14 


99.8% 


99.8% 


99.4% 


15 


100.0% 


100.0% 


100.0% 


16 


100.0% 


100.0% 


100.0% 


17 


100.0% 


100.0% 


100.0% 


18 


100.0% 


100.0% 


100.0% 


19 


100.0% 


100.0% 


100.0% 


20 


100.0% 


100.0% 


100.0% 



5 10 15 20 

Fig. 10 Recognition test via inner product, Hamming and Euclidean measure for (4a) | 



cause oi,j. 
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QUESTION: (3b) J secobj 
ANSWER: (lb) 
NUMBER OF TRIALS: 500 

■ Inner product 

° Euclidean measure 

• Hamming measure 

(right-hand-side questions) 



Agent-object construction. 
[%] recognition 



100% 



50% 



10% 



H \ 1 \ 1 \ 1 \ 1- 

5 10 



15 



H 1 1 1 M 

20 



N 



N 


Hamming 


Euclidean 


Inner pr. 


4 


59.0% 


68.0% 


29.8% 


5 


69.6% 


70.0% 


46.4% 


6 


61.6% 


62.2% 


49.8% 


7 


65.4% 


65.4% 


53.4% 


8 


63.8% 


63.8% 


52.8% 


9 


65.8% 


65.8% 


52.2% 


10 


62.2% 


62.2% 


51.8% 


11 


65.4% 


65.4% 


53.0% 


12 


62.2% 


62.2% 


49.6% 


13 


59.8% 


59.8% 


48.8% 


14 


61.6% 


61.6% 


49.8% 


15 


63.6% 


63.6% 


51.2% 


16 


63.0% 


63.0% 


50.6% 


17 


59.6% 


59.6% 


50.4% 


18 


60.6% 


60.6% 


47.4% 


19 


62.2% 


62.2% 


49.6% 


20 


64.4% 


64.4% 


50.4% 



Agent-object construction with odding blades. 
[%] recognition 



100% 



50% 



10% 



-H 1 1 1 1- 



-N 



N 


Hamming 


Euclidean 


Inner pr. 


4 


46.8% 


75.4% 


26.8% 


5 


88.2% 


92.4% 


40.4% 


6 


88.4% 


93.0% 


45.6% 


7 


95.6% 


96.2% 


66.4% 


8 


98.2% 


98.4% 


78.0% 


9 


99.8% 


99.8% 


83.8% 


10 


99.4% 


99.2% 


90.6% 


11 


100.0% 


100.0% 


93.6% 


12 


100.0% 


100.0% 


91.6% 


13 


100.0% 


100.0% 


93.6% 


14 


100.0% 


100.0% 


94.2% 


15 


100.0% 


100.0% 


95.0% 


16 


100.0% 


100.0% 


92.6% 


17 


100.0% 


100.0% 


95.0% 


18 


100.0% 


100.0% 


93.6% 


19 


100.0% 


100.0% 


92.8% 


20 


100.0% 


100.0% 


93.4% 



5 10 15 20 

Fig. 11 Recognition test via inner product, Hamming and Euclidean measure for (3b) U seeoi,j. 
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QUESTION: (5b) J secotj 
ANSWER: (4b) 
NUMBER OF TRIALS: 500 

■ Inner product 

° Euclidean measure 

• Hamming measure 

(right-hand-side questions) 



Agent-object construction. 
[%] recognition 



100% 



50% 



10% 



10 



15 



H 1 1 1 M 

20 



N 



N 


Hamming 


Euclidean 


Inner pr. 


4 


71.0% 


81.8% 


40.0% 


5 


78.6% 


80.2% 


47.0% 


6 


76.4% 


77.0% 


55.8% 


7 


74.6% 


75.2% 


57.0% 


8 


71.6% 


71.2% 


56.0% 


9 


75.0% 


74.8% 


59.2% 


10 


70.2% 


70.2% 


54.6% 


11 


74.8% 


74.8% 


60.8% 


12 


71.0% 


71.0% 


57.0% 


13 


70.0% 


70.0% 


54.6% 


14 


76.2% 


76.2% 


61.8% 


15 


77.0% 


77.0% 


60.0% 


16 


76.0% 


76.0% 


62.6% 


17 


69.6% 


69.6% 


52.2% 


18 


74.8% 


74.8% 


59.8% 


19 


74.0% 


74.0% 


55.2% 


20 


71.2% 


71.2% 


57.0% 



Agent-object construction with odding blades. 
[%] recognition 



100% 



50% 



10% 



-H 1 1 1 1- 



-N 



N 


Hamming 


Euclidean 


Inner pr. 


4 


61.6% 


84.2% 


39.0% 


5 


90.4% 


91.2% 


46.8% 


6 


89.6% 


90.8% 


52.6% 


7 


96.0% 


95.8% 


54.2% 


8 


97.0% 


97.4% 


58.8% 


9 


98.2% 


98.2% 


64.8% 


10 


99.2% 


99.2% 


55.8% 


11 


100.0% 


100.0% 


61.2% 


12 


100.0% 


100.0% 


57.6% 


13 


100.0% 


100.0% 


58.4% 


14 


100.0% 


100.0% 


62.2% 


15 


100.0% 


100.0% 


55.8% 


16 


100.0% 


100.0% 


61.8% 


17 


100.0% 


100.0% 


58.2% 


18 


100.0% 


100.0% 


61.0% 


19 


100.0% 


100.0% 


58.8% 


20 


100.0% 


100.0% 


57.2% 



5 10 15 20 

Fig. 12 Recognition test via inner product, Hamming and Euclidean measure for (5b) j 
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Agnieszka Patyk 



ing the distance between them, therefore one should avoid calling those measures 
"metrics". 



4.4 Performance of Hamming and Euclidean measures 

In this Section we present some test results comparing the effectiveness of Ham- 
ming and Euclidean measures against the computation of similarity by inner prod- 
uct. These tests are conducted on the data set presented in Table [T] Once the in- 
ner product test indicates more than one potential answer, Hamming and Euclidean 
measures are employed upon the subset of the potential answers — not upon the 



whole clean-up memory. Figures 10 through 12 show test results for sentences with 
various numbers of blades using two types of construction: agent-object construc- 
tion and agent-object construction with odding blades. 

There was no significant difference between results obtained using the agent- 
object construction with odding blades and those obtained with the help of Plate 
construction, therefore results for Plate construction are not presented in the dia- 
grams. Nevertheless, it is more in the spirit of distributed representations to use 
agent-object construction with odding blades since the additional blade is drawn at 
random, whereas the use of Plate construction makes data more predictable. Poor 
recognition in case of the agent-object construction without odding blades results 
from complete or partial similarity cancellation. 

It becomes apparent that the best types of construction of sentences for GA are 
agent-object construction with odding blades and the Plate construction, as they en- 
sure that sentences have an odd number of blades. Further, it is advisable to compute 
similarity by the means of Hamming measure or the Euclidean measure instead of 
the inner product. The Euclidean measure recognizes 100% of items much faster 
(i.e. for smaller data size), but for large data size both measures behave identically. 
Therefore Hamming measure should be used to calculate similarity since its com- 
putation requires less time. The success of those measures is due to the fact that 
the differences between matrices or their signatures lessen the similarity, whereas 
differences in blades did not lessen the value of the inner product considerably. 



5 The Average Number of Potential Answers 

Our point of interest in this Section will be to analyze the influence of acciden- 
tal blade equality on the number of potential answers under agent-object construc- 
tion with appropriate-hand-side reversed questions. The following estimates assume 
ideal conditions, i.e. no two chunks of a sentence are identical up to a constant at 
any time. Intuitively, such conditions could be met for sufficiently large lengths of 
the input vectors, whereas vectors of short length will with high probability be lin- 
early dependent. We will estimate the average number of times that a nonzero inner 
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product comes up when a noisy output is compared with items stored in the clean-up 
memory. We will deal with the following issues: 

How often does the system produce identical blades representing atomic objects? 
How often does the system produce identical sentence chunks from different 
blades? 

How do the two above problems affect the number of potential answers? 

Let V be the the set of multivectors over R'^ stored in the clean-up memory and 
let co{V) be the maximum number of blades stored in a multivector in V. The set of 
all multivectors having the number of blades equal to k is denoted by (Si being 
the set of atomic objects). Naturally, V = SiU-- • U5'(a(v). Let n be a noisy answer 
to some question. Under ideal conditions for every c gV 

\{h\c) \ 7^ n and c share a common blade. (70) 



We will begin with a simple example of a multivector with one meaningful blade 
and L noisy blades. Let ro,...,rihe roles and /o, . . . ,/l be fillers for some L > 0. 
Consider a question 

('•o*/o + '-i*/iH ha*//,) tt ro (71) 

which results in the following noisy answer 

/o-hniH hn/,, ni = r+*ri*fi, 0<i<L. (72) 

Surely, the original answer /o belongs to . Let 5 e V be an arbitrary element of the 
clean-up memory. 

Case 1. Let s € Si and s 7^ /o, in a sense that s might have the same blade as /o 
but is remembered under a different meaning in the clean-up memory. Using basic 
probability methods we obtain 

|(i|(/o-|-niH I-«l))|7^0 s = foOTS = niOT ... OTS = nL, (73) 

P[5' = /oor5' = ni or ... or5' = rtL] = (74) 
Since all blades in Si are chosen independently, the following is true 

£ P[, = /oor. = nior...or. = nz.]= ^'^^'~^^^^+^^ (75) 

Case 2. Let s £ Sk for some 1 < A: < co{V) be a multivector made of k blades, 
i = ii H \-Sk. Since 

P[s does not contain any of {/o,ni, . . . ,«;}] = (1 — (76) 
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2a- 



QUESTION: (lb) ji biteag, 

ANSWER: Fido 

NUMBER OF TRIALS: 1000 

MEANINGFUL/NOISY BLADES: 3/1 

(appropriate-hand-side reversed questions) 



test results 
Equation JTSJ 



-I 1 1 H 



N 



(«) 

N 


(b) 
Eq. f78j 


(0 

test results 


\{b)-{c)\ 


4 


25.9406 


19.626 


6.31459 


5 


16.2396 


12.626 


3.61357 


6 


9.62587 


7.8 


1.82587 


7 


5.61888 


4.736 


0.882881 


8 


3.39396 


2.93 


0.463961 


9 


2.2152 


2.014 


0.205202 


10 


1.6153 


1.473 


0.142299 


11 


1.30909 


1.254 


0.055092 


12 


1.15491 


1.125 


0.029909 


13 


1.07755 


1.057 


0.0205455 


14 


1.0388 


1.026 


0.0127955 


15 


1.0194 


1.021 


0.00159653 


16 


1.0097 


1.006 


0.00370316 


17 


1.00485 


1.0 


0.00485194 


18 


1.00243 


1.003 


0.00057394 


19 


1.00121 


1.0 


0.00121305 


20 


1.00061 


1.002 


0.00139347 



Fig. 13 

ratio. 



5 10 15 20 

Average number of potential answers per 1000 trials with a 1 :3 meaningful-to-noisy blades 



avg number of potential answers 
2a- 



15- 



la- 



5- 



1 



QUESTION: (lb) bite„i,i 
ANSWER: PSmith 
NUMBER OF TRIALS: 1000 
MEANINGFUL/NOISY BLADES: 3/1 
(appropriate-hand-side reversed questions) 



o test results 
■ Equation l|82| 



-I 1 1 1 1 1 H 



N 



H 

N 


(b) 
Eq. <82j 


test results 


\{b)-(c)\ 


4 


25.7459'^ 


19.479 


6.26695 


5 


16.4272 


12.698 


3.72919 


6 


10.1488 


8.175 


1.97385 


7 


6.36 


5.368 


0.992003 


8 


4.25906 


3.785 


0.47406 


9 


3.15034 


2.901 


0.249337 


10 


2.58051 


2.441 


0.139507 


11 


2.29161 


2.249 


0.0426052 


12 


2.14614 


2.136 


0.0101427 


13 


2.07316 


2.045 


0.0281567 


14 


2.0366 


2.032 


0.00459971 


15 


2.01831 


2.021 


0.0026948 


16 


2.00915 


2.015 


0.00584606 


17 


2.00458 


2.004 


0.00057730 


18 


2.00229 


2.003 


0.00071126 


19 


2.00114 


2.0 


0.00114439 


20 


2.00057 


2.0 


0.00057219 



Fig. 14 
ratio. 
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Average number of potential answers per 1000 trials with a 3: 1 meaningful-to-noisy blades 
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2a- 



IS- 



la- 



QUESTION: (4a) (t cause„tj 
ANSWER: fleeag, * Pat + fleeobj * Fido 
NUMBER OF TRIALS: 1000 
MEANINGFUL/NOISY BLADES: 2/2 
(appropriate-hand-side reversed questions) 



o test results 
■ Equation l|85| 



-I 1 1 1 1 1 H 



N 



(«) 

N 


(b) 
Eq. l85l 


test results 


\{b)-{c)\ 


4 


26.1696 


26.908 


0.738392 


5 


17.06 


19.432 


2.37202 


6 


10.9365 


13.65 


2.71353 


7 


7.24505 


9.513 


2.26795 


8 


5.19916 


6.601 


1.40184 


9 


4.11975 


4.924 


0.804253 


10 


3.56505 


4.018 


0.452952 


11 


3.28383 


3.488 


0.204166 


12 


3.14225 


3.246 


0.103753 


13 


3.07121 


3.142 


0.0707938 


14 


3.03562 


3.06 


0.0243762 


15 


3.01782 


3.038 


0.0201829 


16 


3.00891 


3.017 


0.00809016 


17 


3.00446 


3.005 


0.00054476 


18 


3.00223 


3.003 


0.00077229 


19 


3.00111 


3.0 


0.00111387 


20 


3.00056 


3.001 


0.00044306 



Fig. 15 

ratio. 
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Average number of potential answers per 1000 trials with a 2:2 meaningful-to-noisy blades 



avg number of potential answers 
2a- „ 



IS- 



la- 



5- 



1 



QUESTION: (4a) J cause,,,, j 
ANSWER: flee^g, * Pat + flee„tj * Fido 
NUMBER OF TRIALS: 1000 
MEANINGFUL/NOISY BLADES: 2/2 
(right-hand-side questions) 



test results 
Equation l|85| 



-I 1 1 H 



N 



{a) 
N 


{b) 
Eq. ([85|l 


{c) 

test results 


m-{c)\ 


4 


16.\m 


19.815 


6.35461 


5 


17.06 


13.42 


3.63998 


6 


10.9365 


8.854 


2.08247 


7 


7.24505 


5.854 


1.39105 


8 


5.19916 


4.265 


0.934156 


9 


4.11975 


3.411 


0.708747 


10 


3.56505 


2.914 


0.651048 


11 


3.28383 


2.727 


0.556834 


12 


3.14225 


2.645 


0.497247 


13 


3.07121 


2.556 


0.515206 


14 


3.03562 


2.543 


0.492624 


15 


3.01782 


2.513 


0.504817 


16 


3.00891 


2.532 


0.47691 


17 


3.00446 


2.508 


0.496455 


18 


3.00223 


2.495 


0.507228 


19 


3.00111 


2.525 


0.476114 


20 


3.00056 


2.496 


0.504557 



5 10 15 20 

Fig. 16 Average number of potential answers per 1000 trials with a 2:2 meaningful-to-noisy blades 
ratio (right-hand-side questions). 
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we receive the following formula 

^ P [s contains at least one of {/o, «!,...,«/}] = - (1 - ^^^-t^)*). (77) 

Thus, when probing for an answer of /o + ni H L > 0, we are likely to 

receive an average of 

i. 'l^'l-'f"" /£'|s.|(.-(i-^,') ,™, 

potential answers. 

Figure 13 shows test results compared with exact values given by Equation (78 i 
for noisy answers containing one meaningful blade and three noisy blades. Note that 
Equation ( 78 1 is also valid for right-hand- side questions. 

The situation becomes more complex when we are to deal with answers having more 
than one blade. Although items in Si are always chosen independently, we cannot 
say the same about items belonging to S,, I < i < Co{V), since the sentence set is 
chosen by the experimenter. Let us consider the following question 

{bitCagt * Fido + biteobj * PSmith) biteobj (79) 

yielding an answer of four blades 

name * Pat + sex * male + age * 66 -|- bite^i^j * biteagt * Fido . (80) 

Clearly, the correct answer {PSmith) belongs to ^3, but there is one other element of 
the clean-up memory listed in Table[T]that contains a portion of PSmith's blades — 
DogFido 

class * animal + type * dog + taste * chickenlike 
+name * Fido -\- age * 7 + sex * male + occupation * pet, (81) 

the common blade being sex * male. We have two answers that under ideal con- 
ditions will surely result in a nonzero inner product: the correct answer in S3 and 
a potential answer in Sj. By calculations analogous to those leading to Equation 
( 78 1, the average number of answers giving a nonzero inner product for the above 
example is 



41 c, I 4 '»(^) 4 

2+4a^+ L {\Sk\-mi-(l-^f)+ I (82) 

*:G{3,7} k=2,k<^{3J} 



The number of meaningful blades in this example is odd, therefore Equation ( 82 1 is 
also valid for right-hand-side questions. Figure 14 shows test results compared with 
exact values computed by Equation ( 82 1. 
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Let us consider another example — now the question is 

(4fl) U causeobj (83) 

and the answer has a 2:2 meaningful-to-noisy blades ratio 

flecagt * Pcit + fleeohj * Fido + cause^^j^- * [bitCagt * Fido + bite^i,] * Pat) . (84) 

Apart from the correct answer in ^2 ijleeugt * Pat + flecobj * Fido) there are also 
two potential answers belonging to the clean-up memory listed in Table [T| 

• sentence (2b) in — the common blade is flecagt * Pat, 

• sentence (2c) in ^4 — the common blade is flecohj * Fido. 

Therefore, the equation for calculating the estimated number of potential answers 
for this example takes the following form 

4l<f, I 4 , '"(^^ 4 , 

A;G{2.4,8} <r=3,<r^{4.8} 

(85) 

which is illustrated by Figure [TS] 

In this example test results for right-hand-side questions (see Figure [T6| will 



differ from those obtained by formula (85 i by about 0.5. That is because the scalar 



product of (4a) jj cause„i,j and the correct answer will produce two Is which, with 
probability 0.5, will have opposite signs and will cancel each other out. Potential 
answers (2b) and (2c) do not cause such problems, since the number of their blades 
is odd. In half the cases the number of potential answers will be 2 (sentences (2b) 
and (2c)) and in half the cases it will be 3 (sentences (2a), (2b) and (2c)) - achieving 
the average of 2.5 potential answers. 

We are now ready to work out a more general formula describing the average num- 
ber of potential answers for noisy statements with multiple meaningful blades. Let 
S and Q denote the sentence and the question respectively. Let /^^ be the number 
of potential answers to 5 ft 2 in the subset Sk of the clean-up memory V, denote 

by L the number of blades m S Q and p — p\ ^ h p^iy). The formula for 

calculating the estimated number of potential answers to 5 jj 2 then reads 

p^m_p})k + 'f - - (1 - (86) 

provided, that we use appropriate-hand-side reversed questions. As far as right- 
hand-side questions are concerned, this equation may be regarded only as the upper 
bound due to cancellation — for a closer estimate, one should investigate elements 
of the clean-up memory that have an even number of blades. 
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6 Comparison with Previously Developed Models 



The most important performance measure of any new distributed representation 
model is the comparison of its efficiency in relation to previously developed models. 
This Section comments on test results performed on GA, BSC and HRR. 

Naturally, the question of data size arises as a GA clean-up memory item may 
store information in more than one vector (blade), unlike in architectures known 
so far Further, the preferred way of recognition for GA requires the usage of ma- 
trix signatures comprising up to 2 '+^1^71 entries. However, since one only needs 
blades to calculate the matrix signatures, it has been assumed that tests comparing 
efficiency of various models should be conducted using the following sizes of data 

• bits for a single blade in GA, 

• A'A^ bits for a single vector in BSC and HRR, 

where K is the maximum number of blades stored in a complex sentence belonging 
to GA's clean-up memory under agent-object construction with odding blades. For 
the data set presented in Table [T] the maximum number of blades is stored in items 
(3d) and (5b) and is equal to 13. Such an approach to the test data size will certainly 
prove redundant for GA sentences having a lesser number of blades, nevertheless, it 
is only fair to provide relatively the same data size for all compared models. 



Figures 17 through 19 show comparison of performance for GA, BSC and HRR, 
tested sentences range in meaningful-to-noisy blades ratio from 1:2 to 7:2. Clearly, 
GA with the use of Hamming and Euclidean measure ensures quite a remarkable 
recognition percentage for sentences of great complexity and therefore — great 
noise, whereas the HRR model works better for statements of low complexity. There 
is no significant difference in performance of the BSC model as far as complexity of 
tested sentences is concerned. BSC does remain the best model, provided that vec- 
tor lengths for BSC are sufficiently longer than that of GA. Under uniform length of 
vectors and blades GA recognizes sentences better than HRR or BSC, regardless of 
their complexity. 



7 Conclusion 

We have presented a new model of distributed representation that is based on the 
way humans think, while models developed so far were designed to use arrays of 
numbers mainly in order to be easily simulated by computers. 

After a brief recollection of the main ideas behind the GA model, we inves- 
tigated three types of sentence constructions, namely the Plate construction, the 
agent-object construction and the agent-object construction with odding blades. Two 
methods of asking questions were also investigated. As a result, in face of shortcom- 
ings of recognition based solely on the inner product, matrix representation has been 
employed as a recognition tool for the GA model. Using test results computed on 
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QUESTION: PSmith jj name 

ANSWER: Pat 

NUMBER OF TRIALS: 1000 

GA CONSTRUCTION: Agent-object with odding blades, right-hand-side questions 
MEANINGFUL/NOISY BLADES: 1/2 
■ GA, Hamming measure 



" BSC 
■ HRR 



All vector lengths: N. 
[%] recognition 



100% 



50%- 



10%- 



N 


Hamming 


HRR 


BSC 


4 


0.8% 


3.2% 


32.5% 


5 


16.0% 


9.7% 


29.5% 


6 


32.4% 


8.9% 


34.9% 


7 


60.4% 


12.9% 


33.5% 


8 


77.6% 


13.8% 


35.8% 


9 


90.2% 


17.7% 


37.3% 


10 


95.2% 


21.4% 


37.3% 


11 


97.4% 


25.5% 


40.6% 


12 


98.2% 


25.7% 


44.7% 


13 


99.0% 


32.0% 


46.7% 


14 


99.2% 


33.2% 


47.1% 


15 


100.0% 


34.3% 


51.4% 


16 


100.0% 


37.9% 


54.3% 


17 


100.0% 


41.7% 


52.7% 


18 


100.0% 


40.1% 


55.3% 


19 


100.0% 


43.3% 


57.8% 


20 


100.0% 


49.0% 


57.9% 



5 10 15 20 



Vector lengths: N for GA, 13N for HRR and BSC. 
[%] recognition 



100% 



50% 



10% 



-H 1 1 1 1- 



-N 



N 


Hamming 


HRR 


BSC 


4 


0.8% 


90.6% 


92.2% 


5 


16.0% 


95.9% 


97.4% 


6 


32.4% 


97.7% 


98.7% 


7 


60.4% 


98.6% 


99.1% 


8 


77.6% 


99.5% 


99.7% 


9 


90.2% 


99.6% 


99.7% 


10 


95.2% 


99.6% 


99.9% 


11 


97.4% 


99.9% 


100.0% 


12 


98.2% 


99.8% 


100.0% 


13 


99.0% 


100.0% 


100.0% 


14 


99.2% 


100.0% 


100.0% 


15 


100.0% 


100.0% 


100.0% 


16 


100.0% 


99.9% 


100.0% 


17 


100.0% 


100.0% 


100.0% 


18 


100.0% 


100.0% 


100.0% 


19 


100.0% 


100.0% 


100.0% 


20 


100.0% 


100.0% 


100.0% 



5 10 15 20 

Fig. 17 Comparison of recognition for GA, BSC and HRR - PSmith J name. 
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QUESTION: (4a) jj causeobj 
ANSWER: ANSWER: (2a) 
NUMBER OF TRIALS: 1000 

GA CONSTRUCTION: Agent-object with odding blades, right-hand-side questions 
MEANINGFUL/NOISY BLADES: 3/4 
■ GA, Hamming measure 



" BSC 
■ HRR 



All vector lengths: N. 
[%] recognition 



100% 



50%- 



N 


Hamming 


HRR 


BSC 


4 


12.6% 


10.4% 


32.0% 


5 


42.2% 


11.1% 


28.8% 


6 


45.8% 


12.3% 


28.4% 


7 


83.4% 


12.7% 


30.7% 


8 


91.0% 


14.9% 


33.5% 


9 


97.0% 


18.0% 


34.6% 


10 


98.6% 


15.9% 


39.0% 


11 


99.8% 


20.7% 


42.3% 


12 


99.6% 


20.5% 


40.2% 


13 


100.0% 


22.4% 


42.3% 


14 


99.8% 


24.6% 


47.7% 


15 


100.0% 


25.3% 


46.9% 


16 


100.0% 


27.3% 


50.6% 


17 


100.0% 


30.9% 


54.3% 


18 


100.0% 


32.2% 


56.2% 


19 


100.0% 


32.9% 


58.7% 


20 


100.0% 


34.6% 


56.5% 



Vector lengths: N for GA, 13N for HRR and BSC. 
[%] recognition 



100% 



50% 



10% 



-H 1 1 1 1- 



-N 



N 


Hamming 


HRR 


BSC 


4 


12.6% 


75.9% 


92.7% 


5 


42.2% 


82.6% 


95.9% 


6 


45.8% 


86.7% 


98.6% 


7 


83.4% 


92.1% 


99.1% 


8 


91.0% 


94.5% 


99.5% 


9 


97.0% 


96.6% 


99.9% 


10 


98.6% 


97.2% 


100.0% 


11 


99.8% 


97.9% 


100.0% 


12 


99.6% 


98.3% 


100.0% 


13 


100.0% 


98.4% 


100.0% 


14 


99.8% 


99.5% 


100.0% 


15 


100.0% 


99.5% 


100.0% 


16 


100.0% 


99.5% 


100.0% 


17 


100.0% 


99.7% 


100.0% 


18 


100.0% 


99.8% 


100.0% 


19 


100.0% 


99.9% 


100.0% 


20 


100.0% 


99.7% 


100.0% 



5 10 15 20 

Fig. 18 Comparison of recognition for GA, BSC and HRR - (4a) J causCobj- 
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QUESTION: (5a) jj secobj 

ANSWER: (4a) 

NUMBER OF TRIALS: 1000 

GA CONSTRUCTION: Agent-object with odding blades, right-hand-side questions 
MEANINGFUL/NOISY BLADES: 7/2 
■ GA, Hamming measure 



" BSC 
■ HRR 

All vector lengths: N. 
[%] recognition 

100% . . ■ ■ 



N 


Hamming 


HRR 


BSC 


4 


55.2% 


7.6% 


32.9% 


5 


89.4% 


8.4% 


26.1% 


6 


90.0% 


8.6% 


32.5% 


7 


96.2% 


10.2% 


31.0% 


8 


98.0% 


13.0% 


32.9% 


9 


99.4% 


13.4% 


35.1% 


10 


99.0% 


15.6% 


35.3% 


11 


99.6% 


16.6% 


40.4% 


12 


100.0% 


17.5% 


42.6% 


13 


100.0% 


21.1% 


46.0% 


14 


100.0% 


19.7% 


44.8% 


15 


100.0% 


18.8% 


46.5% 


16 


100.0% 


22.2% 


49.9% 


17 


100.0% 


22.3% 


53.8% 


18 


100.0% 


27.3% 


53.6% 


19 


100.0% 


29.0% 


59.7% 


20 


100.0% 


26.6% 


56.4% 



Vector lengths: N for GA, 13N for HRR and BSC. 
[%] recognition 



100% 



50% 



10% 



-H 1 1 1 1- 



-N 



N 


Hamming 


HRR 


BSC 


4 


55.2% 


65.3% 


93.0% 


5 


89.4% 


74.0% 


96.8% 


6 


90.0% 


77.6% 


98.0% 


7 


96.2% 


81.6% 


99.3% 


8 


98.0% 


81.5% 


100.0% 


9 


99.4% 


85.9% 


99.9% 


10 


99.0% 


86.9% 


100.0% 


11 


99.6% 


91.6% 


100.0% 


12 


100.0% 


89.7% 


100.0% 


13 


100.0% 


91.5% 


100.0% 


14 


100.0% 


91.8% 


100.0% 


15 


100.0% 


93.3% 


100.0% 


16 


100.0% 


94.1% 


100.0% 


17 


100.0% 


95.4% 


100.0% 


18 


100.0% 


94.4% 


100.0% 


19 


100.0% 


95.8% 


100.0% 


20 


100.0% 


95.5% 


100.0% 



5 10 15 20 

Fig. 19 Comparison of recognition for GA, BSC and HRR - (5a) J seCobj- 
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a toy model, we have shown that Hamming and EucHdean measures of similarity 
perform very well under the agent-object construction with odding blades. 

We also studied the ways in which the number of potential answers is affected by 
situations in which the system draws at random identical blades denoting different 
atomic objects or in which identical sentence chunks are produced from different 
blades. A formula estimating the number of potential counterparts of a noisy piece 
of information has been derived. Finally, the performance of the GA model has been 
compared with that of BSC and HRR models using sentences of various complexity. 
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